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This review examines a research study that compares high school graduation rates of 
students who used vouchers to attend private high schools in Milwaukee and students 
who attended public high schools in that same city. The study reports that for the most 
recent year of data, a sample of voucher students had estimated graduation rates 12 per- 
centage points higher on average than a sample of public school students. Overall, the 
trend favoring voucher students was observed in five of the previous six years. The anal- 
ysis is technically accurate and makes defensible assumptions as necessary for the final 
calculations. However, although the results are descriptively useful, any real claims about 
whether the voucher program is actually causing higher graduation rates would depend 
on a much stronger research design. 



Summary of Review 




Review 



I. Introduction 

The report reviewed here, Graduation Rates 
for Choice and Public School Students in 
Milwaukee, 2003-2008, is a research brief 
authored by John Robert Warren, a profes- 
sor of sociology at the University of Minne- 
sota/ Dr. Warren wrote the publication on 
behalf of School Choice Wisconsin, a non- 
profit organization that advocates for in- 
creased school choice options.^ The report is 
the third in a series of studies comparing 
high school graduation rates between the 
Milwaukee Parental Choice Program 
(MPCP) and the Milwaukee Public Schools 
(MPS). This particular report adds gradua- 
tion data for 2007-08, the most recent year 
available, to prior years’ calculations dating 
back to 2002-03. 

High school graduation rates have taken on 
a newfound importance in discussions sur- 
rounding school performance and accounta- 
bility. Reports of dismal graduation rates 
imply schools are under-serving large per- 
centages of students, causing increasing 
numbers of high school dropouts, and ulti- 
mately undermining our nation’s economic 
health.^ Best estimates point to a national 
graduation rate of only 70%. Further, low 
graduation rates disproportionally occur in 
poor communities and among students of 
color. ^ 

Within the last several years, concerns have 
been raised over the way the rates have been 
calculated. Increased pressures to look good 
on these measures led to methods of calcula- 
tion that tended to inflate state completion 
rates. ^ In 2008, the U.S. Department of Edu- 
cation began requiring states to use a uni- 
form formula under the No Child Left Be- 
hind Act.^ 



School choice outcomes of interest therefore 
should include graduation rates. The find- 
ings presented in this report show that stu- 
dents enrolled in private schools via the 
Milwaukee Parental Choice Program 
(MPCP) have higher graduation rates than 
students in the Milwaukee Public Schools 
(MPS). 

II. Findings and Conclusions 
OF THE Report 

The study reports that for the most recent 
year of data, a sample of MPCP students had 
estimated graduation rates that were 12 per- 
centage points higher on average than a 
sample of MPS students. After making sta- 
tistical adjustments for ninth grade retention 
and making assumptions about net migra- 
tion, the 2007-08 graduation rate for MPCP 
students was estimated at 77% compared 
with 65% for MPS students. Overall, the 
trend shows MPCP students with higher es- 
timated graduation rates than MPS students 
in five of the previous six years. The report 
concludes that “students in the MPCP are 
more likely to graduate from high school 
than MPS students” (p. 2). It estimates that a 
cumulative total of 3,352 more MPS stu- 
dents would have graduated over the past six 
years had MPS graduated students at the 
same rate as MPCP students. 

III. The Report’s Rationale eor its 
Findings and Conclusions 

The report helpfully discusses the complexi- 
ties and challenges of measuring graduation 

o 

rates. The best methods involve longitudin- 
al monitoring systems, where the enrollment 
status of each student in a state, school dis- 
trict, or school is tracked. However, such 
systems are costly and hard to administer. 
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The analysis here relies on a relatively 
straightforward calculation, what is some- 
times referred to as the “on-time” graduation 
rate. The graduation rate is determined by 
taking the total number of graduates in year 
X and dividing it by the number of students 
who entered the school for the first time as 
ninth graders in year X-4. As noted in the 
report, this is not a perfect measure but is 
intuitively simple and, with certain adjust- 
ments, provides reasonable estimates of 
graduation rates. Ideally, these adjustments 
take into account mortality (whether stu- 
dents die), grade retention, and net migration 
from the school — but these adjustments can 
only be made with data sets containing 
complete information on mortality rates and 
net migration rates among students. For this 
study, those data were not available, but rea- 
sonable assumptions and estimates were 
made to account for migration and grade 
repeaters. 

IV. The Report’s Use oe Research 
Literature 

The report itself is a short research brief and 
as such, devotes little space to a review of 
the research literature. Yet what is reported 
for prior evidence probably puts voucher 
programs in a better light than what is de- 
served. The previous studies cited present a 
one-sided view. To the author’s credit, most 
of the studies cited to show that voucher 
students academically outperform compari- 
son traditional public school students were 
from peer-reviewed journals. But many oth- 
er peer-reviewed studies that have shown no 
effects or negative effects on student 
achievement were not referenced.^ 

V. Review oe the Report’s 
Methods 

The methods used in the report are statisti- 
cally precise and explained in very clear de- 



tail in the appendix. As noted earlier, efforts 
were made to provide good-faith estimates 
of graduation rates for the two groups. A 
number of different estimates were derived 
based upon varying degrees of net migration 
and ninth grade retention within, for the 
most part, empirically supported ranges for 
both groups. For instance, in Appendix Ta- 
ble 3 (p. 12), estimated graduation rates for 
the class of 2008 are presented under differ- 
ent assumptions for net migration (-10%, 
0%, 10%) and ninth grade retention (5%, 
15%, 25%). This allows the reader to see 
how the estimated graduation rates are af- 
fected by assumptions for student enroll- 
ment over the four-year period for each 
group. The result is that under most scena- 
rios, MPCP graduation rates are higher than 
MPS rates. 

Final calculations for the two groups 
represent reasonable estimates for this par- 
ticular four-year graduation rate. For this 
study mortality was assumed to be equal 
across the two groups, a reasonable assump- 
tion. The report also goes to great lengths to 
make suitable statistical adjustments to 
grade retention in the two groups. For the 
final calculations, MPCP program ninth 
grade retention was estimated to be 5% 
based on reports from administrators in 
those schools and 25% for MPS schools 
based on available data from the MPS Of- 
fice of Student Services. These appear to be 
fair adjustments that do not favor one group 
over the other. Finally, net migration was 
assumed to be zero for both MPCP and MPS 
groups. This is an important assumption, 
because if either of the groups have net in- 
or out-migration, this affects the graduation 
rate. For instance, net in-migration has the 
effect of upwardly biasing the rate, while net 
out-migration has the opposite effect. There 
is no evidence presented in the report to 
support the assumption of zero net migration 
for both groups. Without such evidence, it is 
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difficult to tell if this is a safe assumption or 
not. 

VI. Review oe the Validity oe the 
Findings and Conclusions 

The analysis is technically sound and the 
final results are descriptively useful — to a 
point. Any real claims about whether the 
MPCP is actually causing higher graduation 
rates (compared to the MPS) must be based 
on stronger research designs, such as ran- 
domized experiments or causal modeling. 
The report acknowledges this caveat and 
references forthcoming results from a longi- 
tudinal study that looks at two similar 
groups of students in the two treatments. 
The study began looking at ninth graders 
three years ago, so it is too early to learn 
about graduation rates from that cohort at 
this time. 

The drawback of issuing a descriptive report 
such as this is that it may mislead an un- 
trained audience into believing that MPCP 
high schools do a better job of graduating 
students than MPS high schools. As the au- 
thor himself points out, it cannot support 
that conclusion. Moreover, the author’s cau- 
tion is undermined by a key assertion in the 
report. It poses the question, “What if the 
MPS graduate rate in these six years had 
been equal to the rate for high school stu- 
dents participating in the MPCP?” It then 
answers by concluding that a “cumulative 
total of 3,352 additional MPS students 
would have graduated under that scenario.” 
Although mathematically accurate, such 
extrapolation invites causal inferences from 
descriptive data. Given the report’s accurate 
and important caveats about the limits of its 
study design, readers will only be misled by 



this causal conjecture. 

In scientific terms, what’s missing from this 
research design is a “counterfactual.” The 
counterfactual, in this case, would test how 
the MPCP students in the sample would 
have fared in the MPS system. This is why 
we look to study designs with a matched 
comparison group of students (in this case, a 
matched comparison group of students in the 
MPS). There is no matched comparison 
group in this study, which is why the authors 
acknowledge their results cannot proclaim 
the MPCP caused the higher estimated 
graduation rates. 

VII. USEEULNESS OE THE REPORT 
EOR Guidance oe Policy 
AND Practice 

The report offers helpful descriptions of the 
policy context and measurement issues sur- 
rounding graduation rates. This is useful to 
policymakers and educators who may not be 
aware of these matters. The report also pro- 
vides descriptive information about gradua- 
tion rates, which is useful particularly for a 
Milwaukee audience. However, the report 
may confuse readers with its suggestion that 
more MPS students would benefit from the 
MPCP experience. To its credit, the report 
acknowledges that a stronger research de- 
sign is needed to detect private versus public 
school effects for low-income students. It 
references a longitudinal study being con- 
ducted by researchers at the University of 
Arkansas that involves matched pairs of stu- 
dents in MPS and MPCP schools. This type 
of design, if appropriately carried out, is far 
better equipped to address the questions of 
causality that would interest most policy- 
makers. 
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